A Language Independent Scanner Generator

This paper discusses a new methodology for scanner generation that supports language independent lexicon speci cation and automatic generation of stand alone lexical analyzers from speci cations. The mechanism that sits at the basis of this methodology is the layering of the lexicon speci cation on two levels: in the rst level, \universal" lexical constructs which are used as building blocks by most programming languages are de ned, and in the second level, customized lexical constructs of speci c programming languages are speci ed in terms of universal lexical constructs. The universal lexicon is speci ed by regular expressions over a global alphabet used by most programming languages, such as the character set of a keyboard, and is e ciently implemented by deterministic nite automata. The customized lexicon is conveniently speci ed by regular expressions of properties of universal lexical constructs and is implemented by nondeterministic automata whose transition function is determined by the truth value of properties of universal lexemes rather than by the letters of some alphabet. Tools that transform a lexicon speci cation into a stand alone lexical analyzer are developed and reported. The methodology discussed in this paper is convenient because the user operates with meaningful constructs and no programming, in the usual sense, is required. The lexical analyzers thus speci ed are e cient and their correctness is mathematically guaranteed.